This section is intended to introduce various aspects of the art, which may be associated with exemplary embodiments of the present techniques. This discussion is believed to assist in providing a framework to facilitate a better understanding of particular aspects of the present techniques. Accordingly, it should be understood that this section should be read in this light, and not necessarily as admissions of prior art.
Numerical simulation is widely used in industrial fields as a method of simulating a physical system by using a computer. In most cases, there is desire to model the transport processes occurring in the physical system. What is being transported is typically mass, energy, momentum, or some combination thereof. By using numerical simulation, it is possible to model and observe a physical phenomenon and to determine design parameters, without actual laboratory experiments and field tests.
Reservoir simulation is of great interest because it infers the behavior of a real hydrocarbon-bearing reservoir from the performance of a model of that reservoir. The typical objective of reservoir simulation is to understand the complex chemical, physical and fluid flow processes occurring in the reservoir sufficiently well to predict future behavior of the reservoir to maximize hydrocarbon recovery. Reservoir simulation often refers to the hydrodynamics of flow within a reservoir, but in a larger sense reservoir simulation can also refer to the total hydrocarbon system, which may include not only the reservoir, but also injection and/or production wells, surface flow lines, and surface processing facilities. Reservoir simulation calculations in such hydrocarbon systems are based on fluid flow through the entire system being simulated. These calculations are performed with varying degrees of rigor, depending on the requirements of the particular simulation study and the capabilities of the simulation software being used.
The principle of numerical simulation is to numerically solve equations describing a physical phenomenon using a computer. Such equations are generally algebraic equations, ordinary differential equations (ODE), and partial differential equations (PDE). As a means for solving differential equations, ODE and/or PDE, numerically, there are known methods such that the finite difference method, the finite volume method, the finite element method, and the like. Regardless of which method is used, the physical system to be modeled is divided into cells, a set of which is called a grid or mesh, and the state variables that vary in time and in space throughout the model are represented by sets of values for each cell. The reservoir rock properties such as porosity and permeability are typically assumed to be constant inside a cell. Other variables such as fluid pressure and phase saturation may be defined at the boundaries between the cells or at specified points within the cell, sometimes referred to as nodes. A link between two nodes is called a “connection.” Fluid flow between two cells is typically modeled as flow across a boundary between cells or along the connection between the nodes.
A set of equations may be developed to express the fundamental principles of conservation of mass, energy, and/or momentum within each cell and of movement of mass, energy, and/or momentum between cells. The replacement of the state variables that vary in space throughout a model by a finite number of variable values for each cell is called “discretization.” For proper resolution of the modeled physical processes, often hundreds of thousands and even millions of cells are required. That may lead to millions of equations to be solved.
Most reservoir simulators use so-called structured grids in which the cells are assumed to be three-dimensional rectangular shapes distorted to conform as well as possible to geological features and flow patterns. Certain geological features and modeling situations cannot be represented well by structured grids. For example, slanted wells, faults, or fractures that can naturally occur in the reservoir models are almost impossible to model using structured grids. These shortcomings can be overcome in part by using local refinement, in which selected cells are subdivided into smaller cells, and non-neighbor connections, which allow flow between cells that are physically adjacent to each other but are not adjacent in the data structure. A more powerful solution to this problem is to exploit the flexibility provided by layered unstructured grids. In a layered unstructured grid, a computational domain is split into geological layers. In each layer, an unstructured grid is formed by a number of laterally-contiguous irregular-shaped grid cells. The cells forming any layer have corresponding neighbors in the layers located above and below the cell. Such a grid is often referred to as “2.5D grid.” However, layered unstructured grids may not be sufficient to describe the variability of complex geologic structures present in real hydrocarbon reservoirs such as branching wells, complex Y-faults, or pinch-outs that occur when the geological layer disappears inside the reservoir. A fully unstructured three-dimensional grid may provide the flexibility to sufficiently describe complex geological structures. Unlike a layered unstructured grid, a fully unstructured grid is a three-dimensional grid that is unstructured in all three directions.
Due to the large number of calculations, reservoir simulation can require substantial computing resources. Furthermore, a totally unstructured grid may require much greater computational resources compared to a structured grid or a layered unstructured grid. To improve the speed of reservoir simulations, proposals have been made to subdivide a simulation model into smaller segments and to perform computations in parallel on a cluster of multi-processor computing nodes. The principal attraction of parallel computing is the ability to reduce the elapsed time of simulation, ideally by a factor of N for an N-processor computing system. Parallel computing falls short of the ideal because of several factors, including recursion in linear equation solution, the overhead associated with message passing required for various computations, and load imbalances due to heterogeneities in the problem physics and characterization of the hydrocarbon fluids.
Today, most computing systems used in high-performance parallel computing feature a hierarchical hardware design that includes shared-memory nodes with several multi-core processors. Such a system may be referred to as a hybrid, because it combines aspects of a shared memory system with aspects of a distributed memory system. The most commonly used parallelization scheme on hybrid systems uses message passing between the computational nodes and OpenMP programming inside symmetric multiprocessor (SMP) nodes due to the relative easiness of programming. Several academic research papers have been published describing the benefits of such an approach See, for example, (G. Jost, et al., “Comparing the OpenMP, MPI and Hybrid Programming Paradigms on an SMP Cluster”, NAS Technical Report NAS-03-019 (November 2003), available at www.nas.nasa.gov/News/Techreports/2003/PDF/nas-03-019pdf (accessed Jun. 28, 2010); S. W. Bova, et al., “Parallel Programming with Message Passing and Directives”, Computing in Science and Engineering, pp. 22-37 (September 2001); and R. Rabenseifner, “Hybrid Parallel Programming on Parallel Platforms”, The Fifth European Workshop on OpenMP, available at www.compunity.org/events/ewomp03/omptalks/Tuesday/Session7/T01p.pdf (accessed Jun. 28, 2010). The parallelization schemes described in the above references may provide substantial efficiency in cases of homogeneous computational load, for example, when parallelizing loops inside each SMP node. However, in reservoir simulation applications it may be difficult to achieve good speed-up using such an approach due to irregularity of computational loads between the computational nodes and between cores inside the nodes. For application programs with dynamically changing computation and data access behavior, a more flexible programming model that achieves better efficiency may be desirable.
U.S. Pat. No. 7,565,651 suggests a parallel task scheduling system in a multi-threaded computing environment. The patent suggests using one communication thread, one parallel task scheduler implemented in a separate thread, and a plurality of working threads. The scheduler maintains a plurality of task queues, each task queue being dedicated to a particular working thread. The scheduler feeds the task queues with tasks according to a scheduling algorithm. While that scheme may provide much better load balancing of irregular tasks then any one built with the use of OpenMP programming, it may be not flexible enough for task-based parallelism required for reservoir simulation applications. Having just one communication thread and a single task scheduler forces any worker thread to wait its turn to communicate with the scheduler and exchange data with the communication thread. Also, having separate task queues for separate threads easily leads to misbalance of the computational load between the working threads.
In reservoir simulation, several approaches to parallel computations have been developed. The publication W. Liu, et al., “Parallel Reservoir Simulation on Shared and Distributed Memory System”, SPE 64797 (2000), considers three-dimensional distorted rectangular grids on which the reservoir simulation problem is solved in time using pure Message Passing Interface (MPI) communications between the computational SMP nodes and inside the SMP nodes. The paper reported reasonable speed-up on the systems with up to 8 CPUs and fast deterioration of the performance after that limit due to increased communication cost.
Another approach described in U.S. Pat. No. 7,526,418 considers three-dimensional distorted rectangular grids on which the reservoir simulation problem is solved in time. The method is claimed to work on a variety of computer platforms, such as shared-memory computers, distributed memory computers, or personal computer clusters. The approach heavily depends on the assumption of a structured rectangular grid. It splits a computational domain along the x-axis (east-west axis) and distributes the split parts into separate computational nodes using MPI parallelization to exchange the data between those nodes. Inside the nodes, the approach implements OpenMP parallelization along the y-axis (north-south axis). The method described in that patent cannot be applied to the reservoir simulation models defined on 2.5D grids or unstructured grids.
The approach described in U.S. Patent Publication 2006/0235667 can be applied to a structured grid, a 2.5D grid, or the combination of both as long as the resulting grid has layered structure. The method is claimed to work on a variety of computer platforms, such as shared-memory computers, distributed memory computers, or personal computer clusters. The approach heavily depends on the assumption of layered structure of the underlying grid. It splits the computational domain laterally into vertical columns and distributes the split parts into separate computational nodes using MPI parallelization to exchange the data between those nodes. Inside the nodes, the approach can implement OpenMP parallelization. The method described in that patent application cannot be applied to the reservoir simulation models defined on general unstructured grids or use any other type of partitioning of the grid cells between the computational nodes.
All of the reservoir simulation approaches described above employ static partitioning of computational load between computational nodes. However, the variability of complex geologic structures present in real hydrocarbon reservoirs may require consideration of unstructured grids in the presence of heterogeneities in the problem physics and characterization of the hydrocarbon fluids. Different parts of the simulation may require different computational efforts with dynamically changing computation imbalance and data access behavior. Accordingly, a flexible programming model that achieves better efficiency in distributing the computational load of a reservoir simulation may be desirable.